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Abstract 

An  image  articulation  manifold  (IAM)  is  the  collection  of  images  formed  when 
an  object  is  articulated  in  front  of  a  camera.  IAMs  arise  in  a  variety  of  image 
processing  and  computer  vision  applications,  where  they  provide  a  natural  low¬ 
dimensional  embedding  of  the  collection  of  high-dimensional  images.  To  date 
IAMs  have  been  studied  as  embedded  submanifolds  of  Euclidean  spaces.  Un¬ 
fortunately,  their  promise  has  not  been  realized  in  practice,  because  real  world 
imagery  typically  contains  sharp  edges  that  render  an  IAM  non-differentiable  and 
hence  non-isometric  to  the  low-dimensional  parameter  space  under  the  Euclidean 
metric.  As  a  result,  the  standard  tools  from  differential  geometry,  in  particular 
using  linear  tangent  spaces  to  transport  along  the  IAM,  have  limited  utility.  In  this 
paper,  we  explore  a  nonlinear  transport  operator  for  IAMs  based  on  the  optical 
flow  between  images  and  develop  new  analytical  tools  reminiscent  of  those  from 
differential  geometry  using  the  idea  of  optical  flow  manifolds  (OFMs).  We  de¬ 
fine  a  new  metric  for  IAMs  that  satisfies  certain  local  isometry  conditions,  and  we 
show  how  to  use  this  metric  to  develop  a  new  tools  such  as  flow  fields  on  IAMs, 
parallel  flow  fields,  parallel  transport,  as  well  as  a  intuitive  notion  of  curvature. 
The  space  of  optical  flow  fields  along  a  path  of  constant  curvature  has  a  natural 
multi-scale  structure  via  a  monoid  structure  on  the  space  of  all  flow  fields  along  a 
path.  We  also  develop  lower  bounds  on  approximation  errors  while  approximating 
non-parallel  flow  fields  by  parallel  flow  fields. 
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1.  Introduction 


1.1.  Image  articulation  manifolds 

Many  problems  in  image  processing  and  computer  vision  involve  image  en¬ 
sembles  that  are  generated  by  varying  a  small  set  of  imaging  parameters  such  as 
pose,  lighting,  view  angle,  etc.  of  a  fixed  three-dimensional  (3D)  scene.  As  the 
parameters  vary,  the  images  can  be  modeled  as  a  lying  on  a  (typically  nonlinear) 
manifold  called  an  image  articulation  manifold  (1AM)  [1,  2,  3,  4].  Each  point  on 
an  IAM  is  an  image  at  a  particular  parameter  value.  Over  the  past  decade,  there 
has  been  significant  work  [3,  5,  6,  7]  in  learning  and  processing  the  underlying 
geometric  structures  associated  with  image  ensembles.  For  instance,  tasks  such  as 
recognition,  classification,  and  image  synthesis  can  be  interpreted  as  navigation 
along  a  particular  IAM. 

More  specifically,  we  define  an  image  articulation  manifold  (IAM)  as  the  set 
of  images  M  formed  by  the  action  of  an  imaging  map  iona  space  of  articulations 
0,  i.e,  M  =  {i9  =  i(9)  :  9  e  0}.  This  imaging  process  can  be  decomposed 
into  two  steps:  first  the  action  of  the  articulation  on  a  3D  object  or  scene  and 
then  the  subsequent  imaging  of  the  articulated  object/scene  (see  [8]  for  a  detailed 
discussion  of  image  formation).1 

1.2.  IAM  Challenges 

In  spite  of  much  progress,  there  are  fundamental  challenges  to  successfully 
applying  manifold-processing  tools  to  generic  image  data,  in  particular  IAMs. 

First,  it  has  been  shown  that  IAMs  containing  images  with  sharp  edges  are 
non- differentiable  [1].  Specifically,  given  parameters  9\  and  92  with  correspond¬ 
ing  images  Ii  and  I2,  it  follows  that  the  L 2  distance  ||/i  —  I2\\jj  between  images 
Ii  and  1-2  is  a  nonlinear  function  //  (j  1 0 ,  —  9 2\\)  that  is  asymptotically  equivalent  to 
(|| 01  —  02||)5.  Indeed,  jj2  >  c||0i  —  02||~^,  and  this  non-Fipschitz  relation 


'Before  we  proceed  further,  it  is  worth  discussing  certain  degeneracies  in  the  imaging  process 
that  we  wish  to  avoid  in  this  paper  for  analytical  reasons.  In  particular,  we  want  to  avoid  cases 
where  the  set  M  is  not  expressive  of  the  full  range  of  articulations.  As  an  example,  consider  a 
uniformly  colored  3D  sphere  O  undergoing  rotation  about  a  fixed  axis.  Here,  the  parameter  space 
0  is  the  unit  circle  S1.  Being  uniformly  colored,  the  sphere  does  not  change  appearance  under 
rotation,  and  the  IAM  degenerates  to  a  single  point  {Iq}-  Were  the  sphere  richly  textured,  we 
would  obtain  new  views  of  the  sphere  for  each  rotation  so  that  the  IAM  is  isomorphic  to  S1.  For 
the  remainder  of  the  paper,  we  will  consider  IAMs  without  degeneracies  by  assuming  that  the 
imaging  map  *  is  a  re-parametrization  of  0;  i.e.,  we  will  assume  that  the  IAM  is  homeomorphic  to 
the  corresponding  parameter  space. 


2 


Figure  1:  Non-differentiability  of  IAMs  renders  (locally)  linear  transport  inaccurate.  Consider  an 
image  interpolation  task.  Given  images  I\  and  I2,  the  line  connecting  them  a(t)  =  tl\  +  (1  —  t)l2 
is  blurred  and  thus  a  poor  approximation  to  the  geodesic  connecting  R  and  /2. 

indicates  that  the  corresponding  IAM  is  non-differentiable.  Non-differentiability 
suggests  that  local  linear  approximations,  such  as  those  suggested  by  differential 
geometry,  are  invariably  inaccurate  on  IAMs.  To  illustrate  this,  consider  a  stylized 
example  of  image  interpolation  (see  Fig.  1).  Given  images  I\ ,  J2  G  M,  consider 
the  affine  path  a (t)  —  f/i  +  (l  —  f)/2  with  t  G  [0, 1];  for  a  smooth  manifold  (>  C2), 
this  line  would  be  a  close  approximation  to  the  actual  manifold,  especially  over 
small  neighborhoods.  However,  non-differentiability  implies  that  a  first-order  ap¬ 
proximation  to  the  manifold  is  inaccurate  even  over  a  small  neighborhood.  This  is 
illustrated  in  Fig.  1.  To  alleviate  the  non-differentiability  problem,  Wakin  et  al.  [2] 
have  proposed  a  multiscale  smoothing  procedure  that  regularizes  each  point  of  the 
IAM  by  a  set  of  multiscale  Gaussian  smoothing  filters  that  render  the  IAM  smooth 
across  the  various  scales.  This  smoothing  procedure  then  enables  the  definition  of 
linear  tangent  spaces  on  which  one  can  perform  standard  linear  methods  of  analy¬ 
sis.  However,  this  is  unsatisfactory,  since  Gaussian  smoothing  is  inherently  lossy 
—  leading  to  loss  of  high-frequency  information  in  the  images.  Further,  defining 
tangent  vectors  as  the  limit  of  a  multiscale  procedure  is  inherently  complex  and 
moreover  not  possible  for  practical  scenarios  where  we  have  only  samples  from 
the  IAM. 

Second,  conventional  manifold  models  lack  a  meaningful  metric  between 
points  on  the  IAM,  especially  when  the  sampling  of  the  manifold  is  sparse.  Con¬ 
sider  the  simple  translation  manifold  Mt  generated  by  imaging  a  black  disk  of 
radius  R  translating  on  an  infinite  white  background  (see  Fig.  2).  Let  /,  be  the 
image  of  the  disk  with  center  c\  and  J2  the  image  of  the  disk  with  center  c2.  It 
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• 

• 

(a)  Image  /f  (b)  Image  l2 

# 

# 

(c)  Image  l3  (d)  Image  l4 


(e)  Distance  function  ^(||  A C  ||) 


Figure  2:  Euclidean  distance  between  images  on  an  IAM  can  be  meaningless,  (a-d)  Images  of  a 
translating  disk  over  a  white  background,  (e)  The  L 2  distance  between  two  images  depends  on  the 
amount  of  overlap  between  the  the  disks.  However,  the  overlap  is  zero  when  the  distance  between 
centers  exceed  2R\  in  the  figure  above,  d(  l  \ .  I3)  =  d(I  \ .  I4).  Hence,  the  distance  between  two  im¬ 
ages  can  be  written  as  ?7(min(2f?,  ||ci  —  C2 1 1 2 ) )  where  ??(•)  is  a  monotonically  increasing  function. 
Conventional  manifold  learning  techniques,  which  rely  heavily  on  meaningful  local  distances,  do 
not  work  unless  the  sampling  on  the  manifold  is  dense. 


then  follows  that 


h  ~  h\\i?  =  rj(mm{2R,  ||ci  -  c2||)) 


77(||ci  —  c2||),  ||ci  -  c2||  <  2R 
rj(2R),  || ci  —  c2||  >  2 R 


where  rj(-)  is  a  nonlinear  function.  Thus,  when  the  disk  centers  are  separated  by 
a  distance  greater  than  the  diameter  of  the  disk,  the  metric  is  completely  unin¬ 
formative  (see  Fig.  2).  This  suggests  that,  unless  the  sampling  of  images  from 
the  IAM  is  sufficiently  dense,  organization  of  the  images  using  a  construct  such 
as  a  /.'-nearest  neighbor  graph  is  meaningless,  /('-nearest  neighbor  graphs  are  at 
the  heart  of  traditional  manifold  learning  techniques  such  as  LLE  [6],  ISOMAP 
[5],  and  diffusion  maps  [3];  as  a  consequence,  such  techniques  are  doomed  to  fail 
unless  the  sampling  is  sufficiently  dense. 

These  shortcomings  are  exacerbated  for  images  with  rich  textures;  this  rules 
out  a  consistent  analysis  of  IAMs  based  on  classical  differential  geometry  that 
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relies  on  the  smoothness  and  metric  properties  of  the  manifold.  A  critical  missing 
link  in  the  analysis  of  IAMs  is  a  systematic  theory  that  successfully  handles  the 
above  issues  and  permits  new  mathematical  tools  for  IAMs,  including  analogues 
of  traditional  notions  such  as  curvature,  vector  fields,  parallel  transport,  etc.  [9]. 
Clearly,  once  such  analytic  tools  are  available,  we  can  greatly  expand  the  scope  of 
applications  and  pave  the  way  for  efficient  algorithms  specific  to  IAMs. 

1.3.  Transport  operators 

Recently,  a  new  class  of  methods  for  handling  image  ensembles  has  been  de¬ 
veloped  based  on  the  idea  of  transport  operators  [4,  10,  11,  12,  13].  A  transport 
operator  on  an  IAM  is  a  (typically  nonlinear)  map  from  the  manifold  into  itself 
that  enables  one  to  move  between  different  points  on  the  manifold.  Given  im¬ 
ages  /i(x)  and  I2  (x) ,  (where  /(x)  denotes  the  intensity  at  the  spatial  location 
x  =  (x,  y)  G  [0, 1]  x  [0, 1])  a  transport  operator  T  is  a  mapping  that  acts  in  the 
following  fashion: 

/2(x)  =  /1(x  +  T(x)).  (1) 

Instead  of  relying  on  the  linear  tangent  space  that  accounts  only  for  infinitesi¬ 
mal  transformations,  nonlinear  transport  operators  can  be  well-defined  over  larger 
regions  on  the  IAM. 

For  certain  classes  of  articulations,  the  associated  transport  operators  have  al¬ 
gebraic  structure  in  the  form  of  a  Lie  group  [11,  10,  12,  4].  In  such  instances,  it 
is  possible  to  explicitly  compute  transport  operators  that  capture  the  curved  geo¬ 
metric  structure  of  an  IAM.  An  example  of  this  is  the  case  of  affine  articulations, 
where  the  transport  takes  the  form 

/(x)  =  J0(Ax  +  t). 

In  this  case,  the  transport  operator  T(-)  is  of  the  form  T (x)  =  (A  —  I)x  +  t;  this 
can  be  modeled  as  the  group  of  2D  affine  transformations.  The  affine  group  has 
found  extensive  use  in  tracking  [12]  and  registration  [14].  Miao  and  Rao  [10]  leam 
affine  transport  operators  for  image  ensembles  using  a  matrix  exponential-based 
generative  model  and  demonstrate  improved  performance  over  locally  linear  ap¬ 
proximations.  Culpepper  and  Olshausen  [4]  extend  this  framework  using  a  more 
complex  model  on  the  transport  operator  in  order  to  model  paths  on  image  mani¬ 
folds. 

Other  common  examples  of  articulations  in  computer  vision  are  the  projective 
group  (used  to  model  homographies  and  projective  transformations)  and  diffeo- 
morphisms  (used  to  model  ID  warping  functions,  density  functions)  [15].  How- 


5 


ever,  while  algebraic  transport  methods  are  mathematically  elegant,  they  are  appli¬ 
cable  only  to  a  very  restrictive  class  of  IAMs.  Many  IAMs  of  interest  in  computer 
vision  and  image  processing  applications,  including  IAMs  corresponding  to  3D 
pose  articulations  and  non-rigid  deformations,  possess  no  explicit  algebraic  struc¬ 
ture. 

1.4.  Optical  flow-based  transport 

In  this  paper,  we  study  a  specific  class  of  transport  operators  that  are  generated 
by  the  optical  flow  between  images  (we  introduced  this  notion  empirically  in  [13]). 
Given  two  images  I\  and  J2,  the  optical  flow  between  them  is  defined  to  be  the 
tuple  (vx,  vy)  e  L2([ 0,  l]2)  x  L2([ 0,  l]2)  such  that 

h(x,y)  =  Ii(x  +  Yx(x,y),y  +  vy(x,y)).  (2) 

A  common  assumption  in  computing  the  optical  flow  between  images  is  bright¬ 
ness  constancy  [16],  where  the  spatial  intensity  is  assumed  to  not  change  between 
/ 1  and  J2.  Since  the  pioneering  work  of  Horn  and  Schunk  [16],  there  has  been 
significant  progress  towards  the  robust  estimation  of  optical  flow  between  image 
pairs  [17,  18]. 

Optical  flow  is  a  natural  and  powerful  transport  operator  to  transform  one  im¬ 
age  into  another.  In  the  context  of  image  manifolds,  the  collection  of  all  optical 
flow  operators  at  a  point  on  an  IAM  is  a  manifold  of  the  same  dimension  as  the 
IAM  [13].  In  other  words,  at  any  reference  image  on  an  IAM,  there  is  a  corre¬ 
sponding  manifold  of  optical  flow  operators  that  transports  the  reference  image 
along  the  IAM.  This  new  operator  manifold,  which  we  christen  the  optical  flow 
manifold  (OFM),  can  be  used  to  obtain  a  canonical  chart  for  the  IAM  [13];  this 
enables  significantly  improved  navigation  capabilities  on  the  IAM  than  previous 
methods.  In  particular,  OFM-based  transport  is  well-defined  even  in  instances 
when  the  transport  operators  cannot  be  modeled  as  a  Lie  group;  an  example  of 
this  is  the  pose  manifold  (the  IAM  associated  with  rigid  body  motion). 

To  see  the  efficacy  of  optical  flow-based  transport,  consider  again  the  case  of 
image  interpolation,  but  now  with  optical  flow  as  the  transport  operator  (see  Fig. 
3).  Here,  the  path  7 (t),t  G  [0, 1]  generated  on  the  IAM  via  optical  flow  is  a  better 
representative  of  a  path  on  the  IAM,  i.e.,  7 (t)  e  M  for  all  t.  Moreover,  if  the  IAM 
is  generated  via  Lie  group  actions,  then  this  path  coincides  with  the  geodesic. 

As  discussed  earlier,  IAMs  composed  of  images  with  sharp  edges  and  textures 
lack  smoothness  and  hence  do  not  support  locally  linear  modeling.  In  contrast, 
for  a  large  class  of  interesting  articulations,  including  affine  transformations  and 
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Figure  3:  Optical  flow  based  transport  on  an  IAM  leads  to  accurate  image  interpolation.  Consider 
again  the  interpolation  task  in  Fig.  1.  While  locally  linear  transport  on  IAMs  lead  to  inaccurate 
interpolation  results,  the  path  7 (t)  generated  using  optical  flow  provides  an  accurate  approximation 
to  the  true  path  between  L\  and  I2 .  Moreover,  if  the  IAM  is  generated  via  Lie  group  actions,  then 
the  path  generated  coincides  with  the  geodesic. 

3D  pose,  the  corresponding  OFMs  are  smooth  (see  Appendix  A  and  [13])  and 
support  local  linear  modeling.  Moreover,  we  can  define  the  distance  between 
two  optical  flows  to  measure  the  amount  of  motion  required  to  articulate  from 
one  image  to  another.  These  properties  suggest  that  machine  learning  techniques 
(such  as  LLE,  ISOMAP,  etc.)  should  be  able  to  extract  a  considerable  amount 
of  geometrical  information  about  an  image  ensemble  when  applied  to  its  OFMs. 
Figure  4  confirms  this  fact  by  showcasing  the  improvement  in  OFM  vs.  IAM- 
based  dimensionality  reduction  for  an  object  under  2D  translation. 

The  stability  of  OFM-based  modeling  enables  us  below  to  develop  new  dif¬ 
ferential  geometric  tools  for  signal  processing  such  as  the  geometric  mean  and 
Karcher  mean  [19]  of  an  image  ensemble.  The  Karcher  mean  of  a  set  of  images  is 
defined  as  the  image  on  the  IAM  that  is  closest  to  the  set  in  the  sense  of  geodesic 
distance.  Figure  5  shows  the  results  of  Karcher  mean  estimation  using  the  OFM. 
The  interested  reader  is  referred  to  [13]  for  details  of  the  Karcher  mean  estimation 
as  well  as  more  OFM-based  tools  and  applications. 

1.5.  Related  work 

OFMs  fall  under  the  category  of  deformation  modeling,  which  has  been  stud¬ 
ied  in  many  different  contexts,  including  active  shape  [20]  and  active  appearance 
models  [21].  However,  there  are  significant  differences  between  OFMs  and  tra¬ 
ditional  deformation  modeling  approaches.  Morphlets  [22],  for  examlpe,  provide 
a  mutliscale  modeling  of  and  interpolation  across  image  deformations,  but  their 
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(a)  Images 


(b)  Ground  truth 


(d)  OFM 


Figure  4:  Comparison  of  manifold  learning  on  a  IAM  vs.  OFM  (from  [13]).  The  IAM  is  generated 
by  cropping  patches  of  size  200  x  200  pixels  at  random  from  an  image  —  thereby  generating  a  2D 
translation  manifold,  (a)  Sample  images  from  the  IAM  showing  a  few  images  at  various  transla¬ 
tions.  (b-d)  2D  embedding  obtained  using  ISOMAP  on  (b)  the  ground  truth,  (c)  the  IAM,  and  (d) 
the  OFM.  The  near  perfect  embedding  obtained  using  OFM  hints  at  the  near-perfect  isometry  in 
the  OFM  distances. 


treatment  is  limited  to  image  pairs.  In  contrast,  OFMs  apply  to  image  ensembles 
consisting  of  a  potentially  large  number  of  images.  Beymer  and  Poggio  [23]  have 
argued  for  the  use  of  motion-based  representations  for  learning  problems.  How¬ 
ever,  their  goal  is  image  synthesis,  and  therefore  they  offer  no  insights  into  the 
geometric  nature  of  manifold-valued  data.  Jojic  et  al.  [24]  use  a  layered  represen¬ 
tation  to  represent  videos  by  separating  the  appearance  of  moving  objects  from 
their  motion  and  then  representing  each  using  subspace  and  manifold  models, 
respectively.  This  simple,  yet  powerful,  representation  can  model  and  synthesize 
complex  scenes  using  simple  primitives,  but  it  is  not  intended  to  go  beyond  simple 
manifolds  such  as  those  generated  by  translations  and  affine  transformations. 

1.6.  Specific  contributions 

The  main  aim  of  this  paper  is  to  develop  the  mathematical  foundation  of  op¬ 
tical  flow-based  transport  operators  for  image  manifolds,  which  were  introduced 
empirically  [13]. 

We  first  define  a  metric  on  an  IAM  using  its  corresponding  OFMs.  Each  OFM 
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(a)  Sampling  of  images  from  an  IAM 


(b)  Ground  truth 


V — m 


(c)  IAM-based  (d)  OFM-based 

Karcher  mean  Karcher  mean 


Figure  5:  Karcher  mean  estimation  for  20  images  generated  by  the  rotation  of  an  object  about  a 
fixed  axis  [13].  The  images  are  from  the  COIL  dataset  [25].  (a)  Sample  images  from  the  IAM 
showing  a  few  images  at  various  rotations.  We  estimate  the  Karcher  mean  using  local  linear  trans¬ 
port  on  the  IAM  and  the  OFM.  Shown  above  are  (b)  the  ground  tmth  Karcher  mean  of  the  images, 
(c)  the  Karcher  mean  estimated  using  local  linear  transport  on  the  IAM,  and  (d)  the  Karcher  mean 
estimated  using  local  linear  transport  on  the  OFM.  The  accuracy  of  the  estimate  obtained  from  the 
OFM  showcases  the  validity  of  local  linear  transport  on  the  OFM. 


has  a  natural  metric  that  is  a  locally  isometric  function  of  the  corresponding  pa¬ 
rameter  values.  We  consider  the  induced  metric  on  the  IAM,  which  we  dub  the 
flow  metric  and  show  that  the  flow  metric  between  two  points  of  the  IAM  is  a 
measure  of  the  distance  between  the  corresponding  parameter  values.  Next,  us¬ 
ing  the  flow  metric,  we  develop  analytic  notions  of  curvature,  optical  flow  fields, 
and  parallel  transport.  We  analyze  in  detail  the  case  of  optical  flow  fields  defined 
along  a  fixed  curve.  In  particular,  we  define  a  unique  function  associated  to  each 
such  flow  field,  which  we  dub  the  motion  function ,  using  which  we  can  define 
the  notion  of  parallel  flow  fields.  We  answer  the  natural  question  of  how  one  can 
optimally  approximate  a  non-parallel  field  by  a  parallel  field,  and  thereby  induce 
uniform  motion  along  the  curve.  We  also  construct  a  monoid  structure  on  the  set 
of  all  flow  fields  along  a  fixed  curve.  Under  certain  conditions  on  the  curvature  of 
a  curve,  we  show  that  the  space  of  parallel  optical  flow  fields  along  the  curve  is  a 
submonoid  that  comes  with  a  convenient  multi-scale  structure. 

We  envision  that  the  theory  developed  in  this  paper  will  enable  a  large  class 
of  practical  applications  involving  image  manifolds  especially  under  sparse  sam¬ 
pling  of  images  from  the  manifold.  In  this  context,  we  believe  that  the  long  term 
impact  of  this  paper  is  the  first  step  towards  a  complete  theory  of  manifolds  for 
arbitrary  classes  of  signals  using  transport  operators  along  the  lines  of  the  classic 
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differential  geometry  for  smooth  manifolds. 

1.7.  Organization 

The  remainder  of  the  paper  is  organized  as  follows.  In  Section  2,  we  introduce 
OFMs  and,  using  a  fixed  metric  on  the  OFMs,  study  the  induced  flow  metric  on 
the  corresponding  IAM.  We  compare  dimensionality  reduction  techniques  on  the 
IAM  vs.  OFMs.  Using  the  flow  metric,  we  develop  geometric  tools  on  the  IAM  in 
Section  3  and  highlight  their  application  to  parameter  estimation.  We  also  develop 
error  bounds  on  approximating  non-parallel  flow  fields  by  parallel  flow  fields  and 
illustrate  the  idea  with  the  example  of  video  resampling.  In  Section  4  we  develop 
the  multiscale  structure  of  parallel  flow  fields.  We  conclude  in  Section  5  with  a 
brief  discussion. 

2.  OFMs  and  the  Flow  Metric 

In  this  section,  we  define  and  study  the  basic  properties  of  OFMs  correspond¬ 
ing  to  an  IAM  M.  Much  of  the  section  is  concerned  with  a  formal  introduction  to 
OFMs,  first  defined  in  [13],  leading  towards  the  development  of  our  fundamental 
tool,  the  flow  metric  on  an  IAM. 

2.1.  Optical  flow  manifolds 

The  optical  flow  between  two  images  on  an  IAM  measures  the  apparent  mo¬ 
tion  between  the  two  images  and  thus  reflects  the  corresponding  parameter  change 
between  the  two  images.  For  a  fixed  base  image  m  £  M,  consider  a  neighbor¬ 
hood  N(m )  of  m.  If  for  ml  £  N(m)  there  exist  flow  vectors  (vx,  vy)  such  that  ml 
can  be  obtained  from  m  using  the  flow  vector,  i.e.,  m'(x,  y)  =  m(x  +  vx,  y  +  \y), 
then  we  say  that  optical  flow  exists  from  m  to  ml .  We  denote  this  situation  as 
4>vx,vy{rn)  =  ml  or  simply  o{rn)  =  ml .  In  practice,  occlusion  or  boundary  effects 
(i.e.,  veiling  of  certain  portions  of  an  image  due  to  changes  in  the  background 
or  interferes  in  the  scene)  may  lead  to  undefined  estimates  for  the  flow  vectors. 
However,  as  described  in  [13],  one  can  mitigate  these  issues  by  incorporating 
additional  consistency  tests  to  ensure  that  only  the  meaningful  flow  vectors  are 
retained. 

The  set  of  all  points  ml  £  M  for  which  optical  flow  from  m  £  M  to  ml  exists 
is  a  neighborhood  of  m,  which  we  denote  by 

Bm  =  {ml  £  M  :  rri{x,  y)  =  m(x  +  \x,y  +  v„)}. 

Using  this  neighborhood,  we  define  the  optical  flow  manifold  Om  at  m  as  follows. 
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Definition  2.1.  Let  M  be  a  K -dimensional  IAM.  Given  m  £  M,  the  Optical  Flow 
Manifold  Orn  is  defined  as  the  set  of  optical  flows  0  between  m  and  points  in  Brn 

Om  =  {0  =  {vx,vy)  :  4>{m)  £  Bm}. 

The  neighborhood  Brn  is  called  the  flow  neighborhood  around  m. 

()m  is  nonlinear,  i.e.,  an  arbitrary  OFM  is  not  always  a  linear  vector  space.  It 
is  clear  that  the  collection  of  neighborhoods  {Bm  :  m  £  M}  covers  M. 

Proposition  2.2.  Let  M  be  a  K -dimensioned  IAM.  Then  ()m  has  the  structure  of 
a  manifold  homeomorphic  to  Brn . 

Proof  We  first  show  that  Om  is  homeomorphic  to  Brn  and  pullback  the  smooth 
structure  of  Bm  to  induce  a  smooth  structure  on  Om.  Consider  the  map  g  :  Brn  -0 
Orn  that  sends  a  point  m!  £  Brn  to  0  £  Om  such  that  0(m')  =  m.  g  is  clearly 
bijective  and  hence,  we  use  g  to  endow  Om  with  the  quotient  topology.  Moreover, 
since  g  is  injective,  g  is  a  homeomorphism  since  g  is  an  open  map:  given  an  open 
set  U  C  Bm  we  have  U  =  g~1(g(U))  and  by  the  quotient  topology  on  Om,  we 
have  that  V  =  g(U)  is  open  in  Orn  since  g~l(V )  =  U  is  open  in  Brn.  Since 
{Bm  :  m  £  M}  is  a  chart  for  the  manifold  structure  of  M,  there  exists  continuous 
maps  { <I>m  :  m  £  M}  so  that  {(Bm,  (l>rn)  :  m  £  M}  is  an  atlas  for  M.  Therefore, 
the  composition  <f>m  o  g~l  is  a  homeomorphism  from  Om  to  an  open  set  W  in  MA . 
Since  Orn  is  now  covered  by  the  single  chart  {Om,  we  can  pullback  this 

smooth  to  Om  via  °  g~  1  and  endow  Om  with  a  smooth  structure.  Moreover, 
Bm  is  homeomorphic  to  Orn  by  construction.  □ 

We  make  a  few  preliminary  observations.  First,  by  the  homeomorphic  re¬ 
lationship  between  Om  and  Bm,  we  conclude  that  Orn  is  also  a  K -dimensional 
manifold.  Also,  Omi  is  homeomorphic  to  Om2  for  any  mi,m2  £  M.  Moreover, 
the  trivial  element  0O  £  Om  for  each  m  £  M  that  maps  m  to  itself  in  Brn  acts  a 
natural  “origin”  in  Om. 

We  observe,  again  from  the  homeomorphic  relationship  between  Bm  and  Om, 
that  given  any  point  mi  £  Brn  there  is  a  unique  operator  0i  such  that  0i(m)  =  mj. 
A  question  to  ask  is  whether  there  is  any  relationship  between  Om  and  the  tangent 
space  Tm  at  m.  Indeed,  for  a  smooth  IAM,  one  can  define  tangent  spaces  Tm  at 
m  £  M  and  the  OFM  ()m  at  m  is  diffeomorphic  to  a  neighborhood  of  0  £  Tm.  To 
see  this  fact,  recall  that  the  exponentied  map  defined  on  Tm  is  a  diffeomorphism 
between  a  neighborhood  U0  of  0  £  Tm  and  a  neighborhood  of  Vm  of  m.  We  also 
have  a  diffeomorphism  <I>  from  Om  to  Bm  by  definition.  Now,  Bm  is  open  and 
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there  is  an  open  ball  B0  Q  U0  on  which  <h~1  o  exp  is  a  diffeomorphism,  being  a 
composition  of  two  diffeomorphisms. 

As  a  concrete  example,  consider  again  the  translational  manifold  Mt  from 
Section  1 .2.  Note  that  the  parameter  space  in  this  case  is  0  =  M2.  Since  there  is  no 
occlusion  between  any  two  images  on  MT,  it  follows  that  given  any  m,  m'  G  MT, 
there  exists  a  0  G  Om  such  that  0(m)  =  ml.  Since  this  is  true  for  any  pair  of 
images  in  Mt,  we  conclude  that  Om  =  M2  Vm  G  Mt  and  hence,  Brn  =  Mt 
Vm  G  Mt ■  More  generally,  we  note  that  for  an  IAM  M  generated  by  Lie  group 
actions  without  occlusion  between  images,  the  OFM  ()„,  at  any  point  m  G  M  can 
be  identified  with  the  parameter  space  0  and  neighborhood  Bm  at  each  point  is 
the  entire  manifold  M.  In  particular,  one  can  recover  the  geodesic  path  between 
two  any  two  points  m,  ml  G  M  by  using  appropriate  flow  operators  in  Om  to 
generate  the  geodesic  path  from  m  to  ml.  This  shortest  path  corresponds  to  the 
geodesic  in  the  parameter  space  between  the  parameter  values  corresponding  to 
m,  ml  as  well.  We  will  return  to  this  example  in  future  sections  and  show  how  our 
more  general  formulation  contains  the  algebraic  methods  such  as  [11,  10,  12,  4] 
as  special  cases. 

As  in  the  case  of  the  tangent  bundle,  we  can  construct  an  analogous  bundle 
with  the  collection  of  Om  since  m  varies  through  M. 

Definition  2.3.  Let  M  be  an  IAM  and  Om  the  OFM  at  m  G  M.  The  flow  bundle 
OM  on  M  to  be  the  disjoint  union  of  the  Orn  since  m  varies  over  M 

OM  =  ]J  Om. 

meM 

Thus,  an  element  of  OM  is  a  pair  (m,  0)  with  0  G  Om.  Using  this  fact,  we 
can  induce  a  topology  on  the  flow  bundle. 

Proposition  2.4.  Given  a  K -dimensional  IAM  M,  the  flow  bundle  OM  is  a  2/1 - 
dimensional  manifold. 

Proof.  We  first  note  that  if  (m,0)  G  OM  then  {m,  o(rn)}  G  {■ m }  x  Bm.  Thus, 
given  an  atlas  {(U\,  0a)}  as  a  of  M,  we  have  that 

0a,/3(('"T0))  =  (0aM,0/3(0(m))) 

maps  (m,0)  into  f>Q(Ua)  x  LeiUnfj  Brn)  where  Ua  and  U:i  are  charts  around 
m  and  0(m)  respectively.  The  (continuous)  inverse  of  {x\, ...,  Xk,yi,  ...,yk}  G 
f>a{Ua)  X  is  given  by  (m,  0)  where  0Q(m)  =  {xx,  ...,xk}  and  0 

is  the  unique  operator  in  Orn  such  that  wfloitri))  =  {yu ....  yk)  so  that  OM  is 
locally  Euclidean.  □ 
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Thus,  we  see  that  OFMs  are  manifolds  consisting  of  flow  operators  that  are 
defined  pointwise  on  the  corresponding  IAM.  The  action  of  flow  operators  at  a 
base  point  on  the  IAM  results  in  motion  along  the  IAM,  as  opposed  to  linear 
transport  that  results  in  motion  off  the  manifold. 

The  key  property  that  makes  the  study  of  OFMs  interesting  is  that,  for  interest¬ 
ing  IAMs,  the  associated  OFMs  are  smooth  and  exhibit  nice  distance  properties 
[13].  We  summarize  these  in  Appendix  A.  These  two  properties,  namely  smooth¬ 
ness  and  isometry,  are  in  turn  used  to  define  a  meaningful  distance  on  the  IAM. 
We  discuss  this  next. 

2.2.  Metric  structure  on  an  IAM  via  its  OFMs 

Consider  again  the  translational  manifold  Mt,  where  the  OFM  Orn  at  each 
point  m  G  Mt  can  be  identified  with  M2.  Being  isometric  with  M2,  we  can  endow 
each  Om  with  the  Euclidean  metric  which  we  denote  as  d0( •,  •)•  Let  0i,02  €  M2 
be  a  pair  of  parameters  such  that  mi  =  f  (0\  ).  m2  =  i{92);  note  that  there  exists 
a  0  G  Omi  such  that  0(mi)  =  m2.  It  then  follows  that  do{<h,<l>)  =  C\\01-02\\ 
for  some  C  >  0.  As  the  results  of  [13]  indicate,  the  above  discussion  holds 
analogously  for  generic  OFMs,  i.e.,  each  Orn  has  an  associated  metric  do( •,  •)  and 
this  metric  is  locally  isometric  to  a  corresponding  metric  on  the  parameter  space 
0.  We  indicate  this  as 

do((p o,0)  oc  de(9 i,6»2), 

where  0O  is  the  unique  operator  in  Omi  such  that  m i  =  0o(mi). 

Our  main  focus  in  the  remainder  of  this  section  is  to  define  a  corresponding 
metric  for  IAMs  using  the  metric  do(-,-)  on  Om.  The  resulting  metric  on  M 
inherits  the  property  of  being  locally  isometric  to  the  changes  in  parameters.  As 
a  first  step,  we  locally  “push  forward”  the  metric  on  Orn  onto  Brn  as  follows. 
For  points  mi,m2  G  M  with  m2  G  Bmi,  we  have  a  unique  operator  0\  such  that 
rn2  =  0i (mi)  so  that  we  can  define  the  distance  m2)  as  the  corresponding 

distance  between  0O  and  0 i 

dM(mi,m2)  :=do(<j) o,0i). 

Moreover,  if  mi  =  f(0\ )  and  m2  =  f(9 2)  for  parameters  0\ .  d2  G  0  then  we  have 


4(mi,m2)  oc  de(0i,d2). 


However,  this  definition  does  not  readily  extend  to  the  case  where  mi  and  rn2  are 
not  “optically  related”,  i.e.,  m2  0  Bmi.  In  this  case,  we  first  connect  mi  and  m2 
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by  a  path  c  such  that  c(0)  =  mi,  c(l)  =  m2.  We  then  partition  the  domain  of  c  by 
a  partition  P  =  {0  =  t0  <  p  <  ■  ■  ■  <  tn  =  1}  such  that  the  intermediate  points 
along  the  path  are  optically  related  i.e.,  c(tr)  g  B^.  ^,  where  we  assume  that,  for 
a  fine  enough  partition,  we  can  obtain  such  a  nesting.  We  then  define  the  distance 
along  c  to  be 

n—  1 

d(c,mi,m2)  =  supV  dM(c(ti),c(ti+1)), 

P  Z J 

^  i= 0 

where  the  supremum  is  over  all  partitions  of  the  path.By  taking  the  infimum  over 
all  possible  paths,  we  obtain  a  metric  on  M,  i.e., 

n—  1 

4(mi,  m2)  =  inf  V"  dM(c(U ),  c(p+1)).  (3) 

C  Z ' 

2=0 

In  essence,  the  metric  Gpf(v)  is  similar  to  the  Riemannian  distance:  we  first 
define  distance  over  a  fixed  curve  and  then  take  the  infimum  over  all  possible 
paths  (see  Fig.  6). 

Proposition  2.5.  For  an  IAM  M,  the  distance  o?m( •,  •)  in  (3)  is  a  metric  on  M. 

Proof.  Positivity  of  g?m(v)  is  clear,  as  is  the  fact  that  dM(m,m )  =  0. 
If  mi  f  m2,  then  for  every  path  c  between  m\  and  m2,  we  have  that 

n— 1 

^dM(c  (ti),c(ti+ 1))  7^  0  and  hence,  dm^m i,m2)  7^  0.  Symmetry  follows 

i=0 

from  the  fact  that  along  c,  c(p)  G  and  hence,  <Pf(c(p),  c(p+1))  = 

dM(c(U+i),c(ti)).  For  the  triangle  inequality,  we  note  that  given  paths  ci  and 
c2  from  mi  to  m2  and  m2  to  m3  respectively,  the  path  ci  *  c2  obtained  by 
traversing  ci  and  c2  in  succession  at  twice  the  rate  (i.e.  Ci  *  c2{t)  =  Ci(2t)  for 
0  ^  t  ^  \  and  ci  *  c2(t)  =  c2(2t  —  1)  for  |  ^  f  ^  1)  is  a  path  from  mi  to  m3 

n— 1  n— 1  n—  1 

and  ^  dM(ci(fi),  Ci(fm))  +  ^  dM{c2(ti ),  c2(fi+1))  ^  ^  dM(ci  *  c2(p),  Ci  * 
2=0  2=0  2=0 

c2(ti+i)).  By  taking  the  infimum  over  all  such  paths,  we  verify  the  triangle  in¬ 
equality.  □ 

Recall  that  the  metric  on  Orn  satisfies  do(<fi o,0)  oc  de(6 i,d2)  with  0  G  Om 
and  0i,  02  the  parameters  corresponding  to  m  and  0(111)  respectively.  From  the 
above  result,  we  see  that,  with  the  metric  •)  on  M,  we  have 


dM(mi,m2)  oc  de(6 1,02), 
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Figure  6:  A  pictorial  representation  of  the  OFMs  and  the  flow  metric  for  an  IAM.  Each  OFM  is 
mapped  homeomorphically  onto  the  flow  neighborhood  Bm  of  the  corresponding  base  point.  The 
optical  flow  distance  between  two  points  is  computed  by  considering  the  infimum  over  all  possible 
curves  of  the  piecewise  flow  distance  c(tj_|_i)). 


where  mi  and  m2  are  points  on  M  corresponding  to  the  parameter  values  0\  and  d2 
respectively  under  the  assumption  that  the  monotonicity  of  d0(-,  •)  over  different 
{Om}  is  universal,  i.e.,  the  distance  between  two  points  mi  and  m2  does  not 
change  when  we  change  our  definitions  of  flow  neighborhoods.  We  refer  to  this 
metric  on  M  as  the  flow  metric.  We  note  that  the  flow  metric  is  dependent  on  the 
OFMs  Om.  We  can  now  state  the  main  result  of  this  section. 

Theorem  2.6.  Let  M  be  an  IAM  and  let  •)  be  the  associated  flow  metric. 
Then,  dM(tni,m2)  oc  dQ(8i,82)  as  defined  in  (3)  with  mi  and  m2  points  on  M 
corresponding  to  the  parameter  values  0\  and  02  respectively. 

The  smoothness  of  the  OFMs  makes  them  amenable  for  use  with  conventional 
manifold  processing  tools.  For  instance,  common  dimensionality  reduction  meth¬ 
ods  such  as  [5,  6,  7],  etc.  assume  that  the  manifold  is  smooth  and,  hence,  are  tools 
that  are  more  appropriate  for  use  with  the  flow  metric.  As  an  example,  consider 
Fig.  4.  Here,  we  see  that  the  pairwise  flow  metric  between  points  varies  smoothly, 
as  opposed  to  the  Euclidean  metric  on  the  IAM,  where  due  to  sharp  edges,  the 
matrix  of  pairwise  distance  has  large  off-diagonal  entries.  Moreover,  the  resid¬ 
ual  error  of  non-linear  dimensionality  reduction  using  the  flow  metric  decays  very 
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rapidly.  As  a  result,  it  is  more  tractable  to  analyze  IAMs  using  the  flow  metric  as 
opposed  to  the  conventional  Euclidean  metric. 

As  an  example,  consider  the  problem  of  estimating  9  e  0  such  that  i{6)  =  m 
given  a  finite  number  of  template  points  mi,  ■  ■  ■  ,  mn  €  M  with  neighborhoods 
Bmi ,  •  •  ■  Briln  that  cover  M.  As  a  motivating  special  case,  consider  first  the  sit¬ 
uation  when  M  =  MT,  the  translational  manifold  with  0  =  M2  and  a  single 
template  point  ml .  A  similar  problem  has  been  dealt  with  in  [10,  4]  where  the 
authors  estimate  the  Lie  group  generators  corresponding  to  the  IAM.  Given  the 
parameter  value  9'  =  {c\ ,  c2)  of  the  base  image  ml,  we  can  compute  the  the  pa¬ 
rameter  6  corresponding  to  m  as  follows.  First,  we  find  the  unique  flow  operator 
0  =  (0i,  02 )  £  such  that  0(m')  =  m.  The  parameter  value  9  corresponding  to 
m  is  then  obtained  as  9  =  (c[  +  0i,  c2  +  02). 

An  entirely  similar  result  holds  for  generic  IAMs  generated  by  Lie  group  ac¬ 
tions.  From  this  simple  example,  we  see  that  finding  the  optimal  9  is  equivalent  to 
finding  the  optimal  flow  operator  0  that  minimizes  dM(0(m/),  m)  with  0  e  ()m> . 

We  return  to  the  general  case  with  n  template  images  and  seek  the  op¬ 
timal  flow  operator  0  e  Omi  that  minimizes  o?m(0( i  =  1,  •  •  •  ,  n. 
Here,  a  single  neighborhood  does  not  cover  the  entire  IAM,  and,  hence,  to  es¬ 
timate  0  we  first  find  the  neighborhood  Bfh  £  { Brn , ,  •  •  •  Bn,n }  such  that  m  = 
argminm/6{mij...i7Tln}  m').  Our  search  is  then  restricted  to  the  neighbor¬ 

hood  Bm.  Within  this  neighborhood,  we  find  the  optimal  0  e  Or-n  as  above,  i.e., 
0  =  arg  min Tg0^  dM(<j)(rh),  ml).  In  essence,  we  first  find  the  optimal  template 
point  and  then  search  within  the  corresponding  OFM  for  the  optimal  flow  opera¬ 
tor.  If  a  single  template  point  generates  the  entire  IAM,  then  this  procedure  clearly 
reduces  to  the  Lie  group  case  discussed  earlier.  We  thus  see  that  the  generic  OFM 
formulation  includes  the  algebraic  methods  of  [10,  12]  as  a  special  case. 

The  remainder  of  the  paper  is  devoted  to  developing  geometric  tools  for  IAMs 
that  leverage  the  flow  metric.  Note  that,  unlike  the  tangent  space,  the  OFM  has  no 
linear  structure  and,  hence,  we  do  not  have  at  our  immediate  disposal  tools  such 
as  parallel  translation,  covariant  derivatives  etc.  We  will  construct  analogous  tools 
for  our  purposes  via  the  flow  metric  and  hence  open  up  a  vista  for  IAM  analysis. 

3.  Geometric  Tools  for  IAMs  via  the  Flow  Metric 

In  this  section,  we  develop  the  basic  tools  needed  to  analyze  the  structure 
of  IAMs  using  the  flow  metric.  We  will  pay  special  attention  to  flow  operators 
defined  along  curves  on  the  corresponding  IAMs.  Keeping  computations  in  mind, 
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these  tools  will  open  the  door  to  a  variety  of  applications  as  mentioned  in  the 
Introduction  and  beyond. 

3.1.  Flow  radius 

We  first  seek  an  appropriate  measure  of  the  size  of  an  OFM  and  the  corre¬ 
sponding  flow  neighborhood.  In  classical  differential  geometry,  one  measures  the 
radius  at  a  point  in  terms  of  the  injectivity  radius  [9]  using  the  Riemannian  metric. 
In  a  similar  fashion,  we  will  measure  the  radius  of  a  point  m  in  an  IAM  M  in 
terms  of  the  flow  metric. 

3.1.1.  Flow  Radius  at  rri  £  M 

Definition  3.1.  Given  an  IAM  M  and  m  £  M  define  the  flow  radius  or  simply  the 
radius  rm  at  m  as 

rm  =  sup  dM(m,  n). 

neBm 

Note  that  we  may  regard  rm  as  a  function  from  M  to  M+  i.e.  r(m)  =  rm  is  a 
map  from  M  into  the  non-negative  reals.  Moreover,  it  is  continuous  as  a  function 
of  m.  Consider  the  variations  of  rm  as  m  varies.  If  rm  is  large,  then  one  can  find 
a  suitable  operator  £  Om  that  transports  m  to  a  far  away  point,  with  distance 
measured  using  the  flow  metric.  Conversely,  a  small  rm  indicates  that  m  can  only 
be  transported  within  a  small  region,  or  said  differently,  m  obstructs  transport  on 
the  IAM.  Moreover,  rapid  changes  in  the  magnitude  of  rm  within  a  small  neigh¬ 
borhood  indicate  that  the  manifold  is  not  well-behaved  near  m.  In  particular,  this 
indicates  that  there  are  several  points  close  to  m  that  obstruct  transport  while  there 
are  also  several  points  that  allow  flow  over  large  distances  on  the  IAM. 

3.1.2.  Flow  radius  for  Lie  groups 

A  class  of  IAMs  for  which  rm  is  very  well  behaved,  indeed  for  which  rm  is 
a  constant,  are  those  generated  by  a  Lie  group  action.  As  a  motivating  example, 
consider  again  the  translational  manifold  Mt-  We  note  that  as  Om  =  M2  for  each 
m  £  Mt,  it  follows  that  rm  =  oo.  If  the  parameter  space  0  is  compact,  for 
instance,  if  0  =  S1  and  we  consider  affine  rotations  of  a  base  image  m  generating 
the  IAM  M,  then  we  again  have  that  Bm  =  M  and  rm  is  a  constant  0  <  c  <  oo. 
For  a  generic  IAM  M  generated  by  Lie  group  actions  with  Bm  =  M,  it  follows 
that  the  flow  radius  is  a  constant  whose  exact  value  depends  both  on  the  object 
being  imaged  as  well  as  the  nature  of  the  articulation. 
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3.1.3.  Flow  curvature 

The  above  discussion  indicates  that  the  reciprocal  —  is  a  measure  of  “curva- 
ture”.  When  rm  is  large  (or  infinite),  then  the  IAM  is  can  be  thought  as  being 
“optically  flat”  at  m  in  the  sense  that  there  is  no  obstruction  to  transport  on  the 
manifold  at  m.  In  the  other  limiting  case,  i.e.,  if  rm  approaches  zero,  the  only 
operator  in  Om  is  the  trivial  operator  0O  and  hence,  M  is  has  high  “curvature”  at 
m.  We  thus  have  the  following  definition. 

Definition  3.2.  Given  an  IAM  M  and  m  e  M  define  the  flow  curvature  or  simply 
the  curvature  at  m  as  Krn  =  —. 

While  the  traditional  notion  of  curvature  is  a  point  property,  the  flow  curvature 
depends  on  both  the  base  point  as  well  as  its  neighborhood  properties.  In  terms 
of  flow  curvature,  we  can  now  state  that  if  M  is  generated  by  a  Lie  group  then 
M  has  constant  curvature.  The  class  of  IAMs  with  constant  curvature  will  play  a 
prominent  role  in  our  later  analysis. 

3.2.  Optical  flow  fields 

In  this  section,  we  focus  on  a  construct  motivated  by  differential  geometry, 
namely  the  idea  of  a  vector  field  on  a  manifold.  Recall  that  a  vector  field  is  a 
section  of  the  the  tangent  bundle,  i.e.,  a  vector  field  is  a  map  o  :  M  — >•  TM  such 
that  7T  o  a  =  i/hr,  where  t r  is  the  natural  projection  from  the  tangent  bundle  and 
idM  is  the  identity  map  on  M.  In  an  analogous  fashion,  we  define  an  optical  flow 
field ,  or  simply  a  flow  field,  as  a  section  of  the  optical  flow  bundle  of  M. 

While  vector  fields  are  defined  generically  on  manifolds,  the  special  class  of 
vector  fields  along  curves  is  especially  important  in  differential  geometry.  Vector 
fields  along  curves  give  rise  to  tools  such  as  parallel  translation,  Jacobi  fields,  etc. 
[9].  In  the  case  of  IAMs,  which  lack  analytic  structure  in  general,  we  will  define 
optical  fields  along  curves  on  an  IAM  and  recover  similar  geometric  tools  using 
the  flow  metric.  As  we  assign  operators  to  points  on  the  curve,  we  intuitively 
would  like  the  transport  induced  by  the  operator  to  remain  along  the  curve  so  that 
the  collection  of  flow  operators  induces  motion  along  the  curve.  This  special  class 
of  optical  fields  will  be  our  main  object  of  study  for  the  rest  of  the  paper. 

Definition  3.3.  Let  M  be  an  IAM  and  let  c  be  a  smooth  curve  passing  through 
mi,  G  M.  Define  T>c(t)  fl c  =  {c(t')  e  Oc(t)  :  t'  >  t}.  An  optical  field 
from  mi  to  m2  along  c  with  m\,m2  (fi  dc  is  a  map  V  :  t  H »  OM  such  that 
V(t)  Vt  G  Oc(t)  and  Vt(c{t ))  G  Bc(p  f)  c  where  by  Vt(c(t ))  we  mean  the  point 
on  the  curx’e  obtained  by  the  action  ofVt  on  c(t). 
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In  other  words,  an  optical  field  along  c  is  an  assignment  of  a  flow  operator  Vt 
with  Vt  an  element  of  the  OFM  at  c(t)  such  that  such  that  the  action  Vt  on  c(t) 
remains  on  the  curve.  Thus,  the  action  of  Vt  on  the  base  point  c(t )  at  time  t  induces 
motion  along  the  curve.  When  rn \  and  m2  are  clear  from  the  context,  we  simply 
refer  to  Vt  as  the  optical  field  along  c. 

In  essence,  the  curve  Vt  traces  a  curve  in  OM  as  t  varies  with  a  consistent 
action  on  c(t).  By  Bru)  P  c  wc  mean  the  intersection  of  the  K -dimensional  flow 
neighborhood  Bc n\  with  the  one  dimensional  curve  c  starting  at  c(t)  and  hence, 
5c(t)  P|  c  is  a  one  dimensional  embedded  curve  in  B c(q  i.e.,  Bc(t)  P|  c  is  a  one 
dimensional  “slice”  of  Bctty  We  assume  that  Bc(t)  P|  c  is  connected. 

To  measure  the  distance  traveled  by  the  action  of  Vt  on  c(t),  we  define  a  radius 
rt  restricted  along  the  curve  as  opposed  to  the  complete  flow  radius  rc(t)  at  c(t) 

rt  —  sup  dc(c(t),n), 

n^Bc(t)C\c 

where  dc(-,-)  is  the  flow  metric  restricted  to  the  curve  c.  Likewise,  the  curvature 
Kt  along  c  is  the  ratio 


Since  Bc (t)  P|c  is  connected  and  ID,  the  distance  dc{c{t),m)  between  c(t )  and 
m  G  Br(t  )  P|  c  characterizes  m  in  the  following  sense.  Given  any  positive  constant 
0  <  rj  <  rt  there  is  a  unique  m  G  Bc^f]c  with  dc(c(t),m)  =  r).  Thus,  by 
specifying  the  distance  along  the  curve  c,  we  effectively  characterize  the  curve. 
Lifting  this  observation  into  ()cu}.  we  have  the  following  result. 

Theorem  3.4.  Let  M  be  an  IAM,  let  c  be  a  curx’e  passing  through  mi,  m2  G  M, 
and  let  Vt  be  an  optical  field  from  rri  1  to  m2  along  c.  Then,  Vt  is  completely 
characterized  by  the  function  hyit)  =  dc(Vt(c(t)) ,  c{t ))  in  the  sense  that  for  any 
non-negative  function  h(t )  bounded  pointwise  by  rt,  there  exists  a  unique  optical 
optical  field  Vt  such  that  hy(t )  =  h(t). 

Proof  That  0  <  hvU)  <  ‘ft  is  clear  from  the  definitions  above.  Let  hit)  be  any 
non-negative  function  bounded  by  rt.  Then,  for  fixed  /  =  wc  have  that  0  < 
h{t0)  <  rto.  By  the  remark  made  previously,  we  have  a  unique  rnto  G  Bc(U)j  Pc 
with  dc(c(t0),mto )  =  h(t0).  Now,  as  rnto  G  Bc(to)  p  c,  in  particular,  mto  G  Bc{to) 
and  hence  there  exists  a  unique  ot(s  G  Oc(n,)  such  that 


<l>to(c(t0 ))  =  mto. 
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As  t0  was  arbitrary,  as  t  varies,  we  can  define  Vt  =  0t.  Moreover, 

hv{t)  =  dc(Vt{c(t)) ,  c(t))  =  dc((f>t(c(t)),c(t))  =  dc(mt,c(t ))  =  h(t) 
so  that  h(t)  characterizes  V,  .  □ 

Recall  that  we  view  an  optical  flow  field  Vt  along  c  as  a  curve  in  OM.  Theorem 
10  states  that  this  curve  is  characterized  by  the  function  hy(t)  in  the  sense  that  for 
a  given  Vt  along  the  (fixed)  curve  c,  the  function  hv  (t)  contains  all  the  information 
about  the  motion  induced  by  Vt  on  the  curve  i.e.,  hy(t)  measures  the  distance  to 
which  Vt  transports  c(t)  in  Br(t)  f)  c.  Since  this  function  is  of  prime  importance, 
we  make  the  following  definition. 

Definition  3.5.  Given  an  optical  field  Vt  along  a  curx’e  c,  define  its  motion  function 
to  be 

hv(t)  =  dc(Vt(c(t)),c(t)). 

Note  that  by  continuity  of  the  metric  dc(-,  •)  and  c(t),  the  function  hy(t)  is  also 
continuous. 

3.3.  Parallel  flow  fields 

There  is  a  very  natural  geometric  interpretation  of  the  motion  function  hy  of 
an  optical  flow  field  Vt  along  a  curve  c.  Namely,  it  is  a  measure  of  the  distance 
traveled  along  the  curve  at  time  t  by  c(t)  when  acted  upon  by  Vt.  Thus,  it  is  natural 
to  think  of  instantaneous  changes  in  hy  in  t  as  a  measure  of  the  velocity  of  the 
motion  induced  by  Vt  on  c(t).  In  classical  geometry,  the  class  of  constant  velocity 
curves  is  especially  important;  they  correspond  to  uniform  motions.  Similarly,  we 
will  be  interested  in  the  class  of  constant  motion  functions  hy.  These  correspond 
to  optical  fields  Vt  along  c  that  induce  uniform  motion  along  c,  where  by  uniform 
motion  we  mean  the  distance  traveled  along  c  is  constant  for  all  time. 

Another  key  link  with  classical  differential  geometry  is  the  notion  of  parallel 
transport  (or  parallel  translation)  of  tangent  vectors  [9] .  Parallel  transport  is  a  key 
tool  for  “moving”  tangent  vectors  from  different  tangent  spaces  while  preserving 
direction/orientation  along  the  curve.  Parallel  transport  utilizes  the  inherent  linear 
structure  of  the  tangent  space  to  define  a  linear  map  between  tangent  on  points 
along  the  curve.  We  aim  to  develop  a  similar  analytic  tool  for  IAMs.  However, 
an  immediate  stumbling  block  is  the  clear  lack  of  linear  structure  in  the  OFM. 
We  therefore  take  a  different  approach  to  defining  parallel  translation  in  the  OFM 
case  using  motion  functions. 
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To  motivate  our  definition,  we  recall  that  a  vector  field  along  a  curve  c  on  a 
manifold  is  parallel  if  its  covariant  derivative  along  c  vanishes.  The  covariant 
derivative  is  in  essence  a  way  of  differentiating  the  vector  field  along  c.  The  rela¬ 
tion  between  parallel  translation  and  parallel  vector  fields  is  that,  given  a  tangent 
vector  v  in  the  tangent  space  of  a  point  c(t)  on  the  curve  c,  it  is  possible  to  extend 
v  along  c  by  parallel  translation  to  yield  a  parallel  vector  field  along  c.  In  the  OFM 
case,  we  have  the  motion  function  of  an  optical  field  along  c  at  our  disposal  and 
we  use  it  to  characterized  parallelism  of  the  field  along  the  curve. 

Definition  3.6.  An  optical  field  Vt  along  a  curx’e  c,  is  parallel  if  the  derivative  of 
its  motion  function  with  respect  to  t  is  zero,  or  equivalently,  if  the  motion  function 
is  constant  along  c,  i.e.,  hy(t)  is  a  constant. 

In  all  that  follows,  we  will  denote  by  0(c,  mi,m2)  the  space  of  all  optical 
fields  through  mi  and  m2  along  a  curve  c  passing  through  ui\,  m2  G  M.  When 
mi,  are  understood  from  context,  we  simply  refer  to  this  space  as  Q(c).  The 
subclass  of  parallel  fields  along  c  will  be  denoted  by  u(c). 

A  few  facts  are  immediate  from  the  above  definition.  First,  since  hy(t )  <  rt 
for  all  t,  it  is  clear  that  if  Vt  is  to  be  parallel  along  c,  then  hv(t)  is  a  constant  hv 
independent  of  t  and  hy  <  inft  rt.  Therefore,  in  the  future,  we  will  suppress  the 
argument  t  in  the  motion  function  hy(t)  of  a  parallel  flow  field.  Second,  since  Vt  is 
characterized  by  hv,  we  see  that  for  any  constant  5  such  that  0  <  S  <  inft  rt,  there 
is  a  parallel  optical  field  along  c(t)  such  that  hv  =  5.  Such  a  parallel  optical  field 
can  be  obtained,  for  example,  by  choosing  for  each  t  a  flow  operator  <f>t  G  Opt) 
with  dc((j)t(c(t )),  c(t))  =  5.  The  existence  of  such  an  optical  field  ft  is  guaranteed 
by  the  above  theorem.  Given  f  G  0CQ  we  aim  to  extend  0  throughout  the  curve 
to  obtain  a  flow  field  Vt  such  that  Vt  is  parallel  along  c  with  hv  =  dc(fi(c0),  (c0)). 
The  following  result  allows  for  such  parallel  translation  of  flow  operators  along  a 
curve. 

Proposition  3.7.  Let  M  be  an  1AM,  and  let  c  be  a  curve  through  my  m2  G  M. 
Let  e  =  inft  rt.  Then,  given  any  f  G  Ocq0)  with  5  =  dc(f(c(t0)) ,  c(t0 ))  <  e  there 
exists  a  unique  parallel  optical  field  along  c(t )  with  hy(t)  =  5. 

Proof  As  5  <  e,  in  particular  5  <  rt.  Thus,  invoking  the  previous  theorem  for  the 
special  case  of  the  constant  function  h(t)  =  5,  we  have  the  existence  of  a  unique 
optical  field  Vt  along  c(t)  such  that  hy(t)  =  5.  Moreover,  since  hy(t)  is  constant, 
Vt  is  parallel.  □ 
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In  contrast  with  the  classical  case,  parallel  transport  along  a  curve  is  dependent 
on  the  nature  of  the  flow  operator,  i.e.,  an  arbitrary  flow  operator  0  £  Oco  cannot 
be  parallel  translated  along  c  unless  dc(0(co),  c0)  <  inf trt.  This  constraint  is 
related  to  the  nature  of  the  curve;  parallel  transport  along  a  curve  that  contains 
points  with  high  curvature  Kt  is  limited  to  those  flow  operators  that  induce  smaller 
motion  along  the  curve.  Moreover,  the  possibility  of  parallel  transport  of  0  G  Oc(t) 
has  a  global  dependence,  i.e.,  it  depends  on  the  curvature  of  the  entire  curve,  not 
only  the  curvature  Kt  at  the  point  c(t). 

Thus,  we  are  naturally  led  to  study  those  curves  for  which  parallel  translation 
of  an  operator  0  at  a  single  point  c(f0)  ensures  the  existence  of  parallel  translation 
of  operators  at  any  other  point  on  the  curve.  Clearly,  the  necessary  condition  is 
the  invariance  of  Kt  with  t,  and  therefore  we  are  led  to  consider  curves  for  which 
the  curvature  Kt  is  independent  of  t,  i.e.,  a  constant.  This  special  class  of  curves 
has  a  very  rich  structure  that  we  explore  in  the  following  sections. 

3.4.  Approximation  of  arbitrary  flow  fields  by  parallel  flow  fields 

In  this  section,  we  consider  the  problem  of  approximating  an  arbitrary  Vt  £ 
fl(c)  by  elements  in  u(c).  Consider  an  optical  field  Vt  £  £l(c)  (not  necessarily 
parallel)  along  a  curve  c  of  constant  flow  curvature.  A  natural  question  to  ask 
is  how  far  away  Vt  is  from  being  parallel.  One  way  to  do  this  is  to  seek  the 
“best”  approximation  of  10  by  a  parallel  field  Wt  £  u(c)  along  c.  To  quantify  the 
approximation,  we  consider  the  following  cost 

ef)  =  dc(Vt(c(t)),Wt(c(t))). 

In  words,  e(t)  is  a  measure  of  how  the  action  of  Vt  on  c(t)  differs  from  the  action 
of  Wt  on  c(t).  Note  that  e{t)  is  bounded  above  by  hy{t )  +  hw  since  e(t)  = 
dc(Vt(c(t)),  Wt(c(t )))  <  dc(yt(c(t)),  c(t ))  +  dc(c(t),  Wflc{t)))  =  hv(t)  +  hw. 

While  e(t)  is  a  pointwise  error,  we  will  need  to  consider  the  total  error  over 
the  entire  curve.  To  this  end,  a  natural  choice  of  error  metric  is 

E(V,W)=  f  e(t)dt, 

J  a 

where  the  domain  of  c  is  the  interval  (a,  b ).  Thus,  our  goal  is  to  find  a  parallel 
optical  field  Wt  £  u(c)  that  minimizes  E(V,  W),  i.e., 

W*  =  arg  min  E  ( V,  W) . 
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In  general,  a  minimizer  may  not  exist,  or  it  may  not  be  unique  if  one  exists.  How¬ 
ever,  the  greatest  issue  is  the  strong  dependence  of  the  error  e(t)  on  the  flow 
metric,  which  prevents  a  generic  solution  to  the  minimization  problem  since  we 
cannot  infer  the  convexity  of  the  problem  as  stated.  We  can,  however,  obtain  a 
universal  lower  bound  on  the  error  independent  of  the  flow  metric  as  follows 


I hv{t)  -  hw |  =  | dc(Vt(c(t)),c(t))  -  dc(W(c(t)),c(t))\  <  e(t). 


Therefore,  we  seek  a  minimizer  of  /  \hv(t)  —  hw\dt.  Note  that  since  Wt  is 

J  a 

parallel,  hw  is  a  constant,  say  h  >  0.  Moreover,  since  c  is  a  path  of  constant 
flow  curvature,  rt  is  a  constant  r  >  0.  Since  h  characterizes  Wt,  a  minimizer  h* 
yields  a  lower  bound  for  E(V,  W).  Our  goal  then,  is  to  find  an  optimal  constant 


h*  that  minimizes  E(h)  =  /  \hy{t)  —  h\dt.  We  claim  that  a  solution  h*  is  a 


certain  “median”  of  hv(t).  We  first  define  Ak  =  {t  €  (a,  b)  :  hv(t )  >  k}  and 
Bk  =  {t  €  (a,  b )  :  hy(t)  <  k}  for  some  k.  We  claim  that  an  optimal  constant  h 
is  such  that  A (A~h)  =  A (B~h),  where  A(S')  denotes  the  measure  of  a  set  S. 


Theorem  3.8.  Let  h  be  the  constant  such  that  Af/l^j  =  X(Bh  ).  Then,  h  minimizes 

m- 


Proof  Note  that  the  function  E{h)  is  convex  in  h  with  h  £  (a,  b )  and  (a,  b )  a 
convex  interval.  Therefore,  we  are  guaranteed  a  minimizer  h*.  Now,  without  loss 
of  generality,  we  assume  that  h*  <  h.  We  evaluate  the  cost  over  the  two  regions 

Ah  and  Bh 


E{h*)  =  /  |  hy{t)  —  h*\dt  =  /  \hv{t)  —  h*\dt+  /  \hy{f)  —  h*\dt. 


Define  Bi  —  {t  :  h*  <  hv{t )  <  h }  and  B2  —  {t  :  hy(t )  <  h*}  and  note  that 
A (A-f)  —  X(Bi)  —  X (B2)  =  0.  We  can  now  express  E(h*)  in  terms  of  E{h)  as 

E(Jd)  =  E^(h)  +  (h  -  h*)X(A%)  +  E^ffi)  -(h  -  h*)X(B2)  -  a, 

where  Ex(h )  denotes  the  cost  function  restricted  to  the  subset  X  C  (a,  b)  and  a 
is  a  positive  constant  that  measures  the  difference  of  h  —  h*  on  the  set  Bx.  The 
maximum  value  of  a  is  (h  —  h*)X(Bi)  and  hence 

Em  <  Ea$)  +  (h  -  h*)(X(A~h)  -  A (B1)  -  X (B2))  +  E^(h). 
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Since  X(A^)  —  X(B1)  —  X(B2)  =  0,  we  conclude  that 

E{h*)  <  E(h ) 

and  hence  h*  =  h.  The  case  of  h*  >  h  follows  from  symmetry.  □ 


We  illustrate  the  approximation  of  non-parallel  flow  fields  by  parallel  flow 
fields  with  the  example  of  video  resampling  where  we  can  consider  a  video  X  to  be 
a  curve  on  an  IAM  i.e.,  a  video  X  =  {/t ,  0  <  t  <  T}  with  T  >  0.  This  application 
is  related  to  the  problem  of  dynamic  time  warping  (DTW)  [26,  27,  28],  where 
one  is  interested  in  measuring  the  similarity  between  two  sequences  that  vary  in 
time  or  speed.  As  we  shall  see,  this  can  be  used  for  matching  or  aligning  video 
sequences  with  a  warped  time  axis  [27,  28]. 

Consider  the  IAM  generated  by  imaging  a  black  disk  on  an  infinite  white 
background  starting  with  an  initial  velocity  v0  and  accelerating  with  constant  ac¬ 
celeration  a  along  a  fixed  direction.  For  instance,  the  disk  can  be  thought  of  as 
undergoing  freefall  off  an  infinitely  high  cliff.  The  IAM  is  a  ID  curve  c  and 
homeomorphic  to  M+.  the  non-negative  reals.  Note  that  the  curvature  is  every¬ 
where  zero  since  rt  =  oo.  Given  an  arbitrary  flow  field  Vt  along  c,  our  goal  is 
to  analytically  construct  a  parallel  field  Vt  such  that  Vt  is  the  unique  parallel  flow 
field  that  minimizes  E(h).  We  first  consider  the  video  obtained  by  the  action  of  Vt 
i.e.  X  =  (I4(c(f))}.  Since  Vt  is  not  parallel,  the  video  will  show  the  disk  moving 
with  non-uniform  motion.  Our  goal  is  to  make  the  video  uniform,  i.e.,  generate  a 
new  video  X  from  V,  that  shows  the  disk  moving  with  uniform  motion. 

From  the  physics  of  the  problem,  it  is  clear  that  dc(c(t),  c(t  +  A, ) )  =  K(vt8t  + 
-/id'l)  for  some  positive  constant  K  and  any  time  increment  8t  with  vt  being  the 
velocity  at  time  t.  Now,  since  Vt  is  to  be  a  parallel  flow  field,  we  have  that  hy  is 
a  constant  denoted  by  h.  Thus,  dc(Vt(c(t)),  c(t ))  =  dc(c(t),c(t  +  8t))  =  h  so  that 
K(vt8t  +  -/idj)  =  h.  Rearranging  this  equation,  we  arrive  at 


2  vt 
a 


r2  .  r 

°t  d - - T7 


2  h 
a. K 


0. 


Solving  for  8t,  we  obtain  two  real  roots  ±  y  (a)2  +  ^|  of  which  the 

(physically  meaningful)  positive  root  5]  —  ^  +  \J (^)2  +  ^  >  0  is  retained. 
Thus,  by  defining  Vt  such  that  Vt(c(t ))  =  c(t  +  8}),  we  see  that  Vt  is  the  unique 
parallel  flow  field  that  minimizes  E(h).  The  new  video  X  =  { Vi(c{t))}  will 
thus  show  the  disk  moving  with  constant  velocity.  With  this,  we  have  effectively 
linearized  the  motion  and  made  it  independent  of  the  acceleration  of  the  disk. 
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4.  Multiscale  Structure  of  Parallel  Flow  Fields 

As  indicated  in  the  Section  3,  the  set  of  parallel  optical  fields  is  a  very  special 
subset  of  the  set  of  all  optical  fields  along  a  fixed  curve  c.  In  this  section,  we 
will  construct  a  monoid  structure  (i.e.,  a  set  with  an  associative  operation  and 
identity)  on  the  set  of  all  optical  fields  along  a  fixed  curve  c  and  show  that  the 
class  of  parallel  fields  forms  a  submonoid  of  this  set  under  some  conditions  on 
the  curvature  along  the  curve.  Moreover,  the  monoid  operation  yields  a  multiscale 
structure  on  the  set  of  parallel  optical  fields. 

4.1.  Monoid  structure  on  13(c) 

As  noted  previously  in  Section  2,  a  clear  disadvantage  in  dealing  with  the 
space  of  all  optical  fields  along  a  curve  is  the  lack  of  a  linear,  or  more  generally, 
any  algebraic  structure.  In  order  to  remedy  this  situation,  we  will  define  a  binary 
operation  on  the  set  of  optical  fields  that  yields  a  monoid  structure.  We  first  fix  a 
curve  c  passing  through  mi,m2  G  M. 

Recall  that  a  generic  Vt  G  13(c)  is  characterized  by  its  motion  function  hy{t). 
Thus,  operations  defined  on  motion  functions  hy(t)  translate  to  operations  on 
Vt  G  13(c).  With  this  in  mind,  we  define  for  Vt,  Wt  G  13(c)  the  sum  Vt  +  Wt  to  be 
the  unique  optical  field  with  motion  function 

hv+w(t )  =  min  (hv(t)  +  hw(t),rt). 

Since  hy+w(t)  <  r,  for  all  t,  we  see  that  hy+w(t )  corresponds  to  a  unique  flow 
field  that  we  define  to  be  Vt  +  Wt.  This  operation  is  clearly  commutative.  Note 
also  that  the  trivial  (parallel)  field  Zt  defined  to  be  the  field  that  acts  trivially  on 
c(t);  i.e., 

zt(c(t))  =  c(t) 

is  characterized  by  the  motion  function  hz(t)  =  0  since  hz{t)  = 

dc(Zt(c(t),  c(t))  =  dc(c(t),c(t ))  =  0.  Moreover,  for  any  Vt  G  13(c),  we  have 
that 

zt  +  vt  =  vt. 

We  see  that  Zt  acts  as  the  identity  element  in  13(c).  In  addition  to  this,  there  is  also 
the  unique  optical  field  Ut  characterized  by 

hu(t)  =  r  t 


that  satisfies 


Vt  +  Ut  =  Ut 
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for  all  \ /(  £  fi(c)  and  hence  acts  as  the  “absorbing”  element  of  0(c). 

Clearly,  we  do  not  have  “inverses”  with  respect  to  “+”  in  the  sense  that,  given 
a  generic  Vt  G  0(c),  there  does  not  exist  a  Wt  such  that  Vt  +  Wt  —  Zt.  However, 
we  do  have  “conjugates”  with  respect  to  “+”  in  the  following  sense.  Given  any 
Vt  G  0(c),  there  is  a  unique  V*  G  0(c)  such  that 

C*  +  V*  =  Ut. 


Vt*  is  defined  by  its  motion  function 


hv(t)  =  rt  —  hv(t). 

We  refer  to  Vt*  as  the  conjugate  of  Vt.  Moreover, 

(v;r  =  v,. 

Finally,  we  note  that  Ut  and  Zt  are  conjugates. 

Proposition  4.1.  0(c)  is  a  monoid  under  the  operation  “+  ”  defined  in  (22). 

Proof.  We  only  verify  associativity,  since  Zt  provides  the  identity.  Given 
Vt,Xt,Yt  G  0(c),  we  consider  the  sums  (V,  +  Xt)  +  Yt  and  V,  +  (Xt  +  Yt). 
If  hv(t),  hx(t),  hY(t )  are  such  that  hv(t)  +  hx(t)  +  hY(t )  <  rt  then  both  sides 
are  clearly  equal.  If  on  the  other  hand,  hv(t)  +  hx(t)  +  hY(t)  >  rt  then  we  must 
have  that  either  the  sum  hv(t)  +  hx(t)  +  hY(t)  taken  two  factors  at  a  time  exceeds 
rt  or  that  the  combined  sum  of  all  three  factors  exceeds  rt  with  the  sum  of  no  two 
factors  exceeding  rt.  In  the  first  case,  we  assume  that  hY(t)  +  hx(t)  >  rt,  which 
implies  the  sum 

(Vt  +  Xt)  +  Yt  =  Ut 

with  (Xt  +  Yt)  f  Ut,  i.e., 


hx(t)  +  hY(t )  <  rt. 

However,  Vt  +  (X,  +  Yt)  is  characterized  by 

min  (hv(t)  +  min  (hx(t)  +  hY(t),rt),rt)  =  min  (hv(t)  +  hx(t)  +  hY(t),rt )  =  rt 

which  shows  that 


Vt  +  (Xt  +  Yt)  =  Ut 

as  well.  The  other  cases  follow  by  similar  arguments. 


□ 
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4.2.  Parallel  fields  along  cun’es  of  constant  cun’ature 

The  addition  operation  defined  above  restricts  to  the  set  u(c)  of  parallel  fields 
along  c(t).  However,  for  generic  c(t),  the  sum  Vt  +  Wt  of  two  parallel  fields 
1 /t,Wt  G  tc(c)  may  result  in  a  possibly  non-parallel  field.  For  instance,  consider 
hv(t),hw(t)  such  that  hv(t0 )  +  hwif o)  <  n0  but  hy(t i)  +  hwifi)  >  rtl  for 
some  t0,ti.  Clearly,  the  sum  is  not  parallel  since  in  the  first  case,  hv+wiUi)  = 
hv(t0 )  +  hw(t0 )  <  rto  while  hv+w(ti)  =  rtl  so  that  hv+w(t )  is  not  constant. 

However,  that  if  rt  is  constant  along  c,  i.e.,  the  flow  radius  is  constant  along 
the  path,  then  the  above  situation  is  vacuous.  Since  Kt  =  ^is  the  curvature  along 
the  curve  c,  we  are  essentially  requiring  the  curvature  to  be  constant  along  c.  We 
formally  record  the  above  observation. 

Proposition  4.2.  Let  c  be  a  curx’e  with  constant  cun’ature  Kt.  Then,  the  operation 
“+  ”  restricted  to  uj(c)  is  well  defined  and  u(c)  is  a  submonoid  offl(c). 

Proof.  We  need  only  verify  closure  of  “+”  in  cc(c)  since  Zt  e  t c(c).  If  Vt  G  t c(c), 
then  hv(t)  is  a  constant  and  we  will  therefore  suppress  the  argument  t  in  hy(t). 
Note  that  since  c  is  of  constant  curvature,  rt  is  a  constant  r  >  0.  Thus,  if  Vtl  Wt  G 
tc(c),  then  hy+w  =  min (hy  +  hw,  r )•  If  hy  +  hw  <  r  then  as  hy  and  h\y  are 
both  constant  so  is  their  sum  hv  +  hw  and 

lly+W  =  hy  +  h\y. 


If  hy  +  hw  >  r,  then 


hv+w  =  min  (hy  +  hw,r)  =  r. 


In  either  case,  hv+w  is  constant  and  hence 

Vt  +  Wt  Gw(c). 


□ 

Thus,  curves  c  that  have  constant  curvature  at  all  points  are  very  special;  not 
only  is  parallel  transport  determined  by  a  single  point  on  the  curve,  but  tc(c)  is 
also  a  submonoid  of  f2(c). 
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4.3.  Multiscale  structure  of  u(c) 

In  this  section,  we  look  for  finite  submonoids  of  c o(c).  We  would  like  the 
submonoids  to  be  canonically  defined,  by  which  we  mean  that  they  depend  only 
on  the  geometry  of  the  constant  curvature  curve  c.  In  particular,  we  construct  for 
each  positive  integer  k,  a  collection  of  finite  submonoids  Vn,k  of  co(c)  with  n  a 
non-negative  integer  and  c  a  constant  curvature  path.  Moreover,  we  will  see  that 
this  collection  is  naturally  nested,  i.e., 

V0, A;  C  Vi,fc  C  •  •  • 

In  essence,  this  provides  a  multi-scale  view  of  c e(c). 

Since  ce(c)  consists  of  parallel  optical  fields,  we  will  simply  denote  the  motion 
function  hv(t )  of  Vt  G  u(c)  as  hy.  Moreover,  we  will  suppress  the  subscript  t 
when  referring  to  elements  of  Vt  G  cj(c).  Also,  for  V  G  u(c),  we  mean  by  j  the 
element  in  u(c )  with  motion  function  being  the  constant  Recall  also  that  uj(c ) 
possesses  two  canonical  elements  Z,  U  G  co(c)  D  f2(c)  that  act  as  the  trivial  and 
absorbing  elements  of  u(c )  respectively. 

We  begin  now  with  the  construction  of  Vnj-.  Fix  a  positive  integer  k  and  set 

Vo ,k  =  {Z,  U}. 


Next,  we  inductively  set 


Vn,fc 


7  U_  2U 


( kn-l)U 


,U 


Now,  it  is  clear  that 


V0,fc  c  Vi ,k  c  .... 

Moreover,  by  the  constant  curvature  condition,  the  sum  of  any  two  elements  in 
Vn.k  remains  in  V,,./,  while  associativity  is  obtained  from  the  corresponding  prop¬ 
erty  in  u(c).  Finally,  since  Z  G  V,,  j.  for  all  a.  k,  we  conclude  that  each  V„,/.  is  a 
finite  submonoid  of  c e(c). 

We  have  thus  obtained  a  sequence  of  finite  submonoids  of  ce(c).  Moreover, 
with  increasing  n,  it  is  clear  that  an  arbitrary  V  G  uj(c)  can  be  uniformly  approx¬ 
imated  by  V  G  V,,./,  in  the  sense  that  hy  —  hy\  can  be  made  arbitrarily  small  by 
choosing  larger  n.  In  other  words,  the  sequence  Vn±  is  “dense”  in  ce(c).  Finally, 
since  the  basic  generators  of  V,,./,  are  Z.  U  G  cu(c)  D  fl(c),  this  construction  is 
canonical  in  the  sense  that  it  depends  only  on  the  set  of  {Z,  U},  which  are  in  turn 
are  characterized  by  the  global  geometry  of  the  curve  c. 
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In  Section  3,  we  saw  that  with  curvature  conditions  on  a  curve  c,  it  is  possible 
to  approximate  an  arbitrary  flow  field  by  a  parallel  one.  The  results  presented 
above  show  that  one  can  further  approximate  a  parallel  flow  field  with  a  finite 
collection  of  “template”  elements  from  u>(c).  This  is  very  much  similar  to  the 
multiscale  representation  that  wavelets  provide  for  natural  images.  Indeed,  the 
multiscale  structure  inherent  to  monoids  paves  the  way  for  “lossy  compression” 
of  arbitrary  flow  fields  by  storing  only  the  relevant  scales  Vn,fc.  In  addition  to 
compression,  the  multiscale  structure  can  potentially  enable  fast  computations  on 
the  flow  fields.  Operations  on  the  flow  fields  can  equivalently  be  mapped  to  those 
on  the  Vn  k  without  loss  in  accuracy  while  gaining  significantly  in  the  number  of 
computations  required. 

5.  Discussion 

In  this  paper,  we  have  developed  the  mathematical  foundations  of  optical  flow- 
based  transport  operators  for  image  manifolds,  which  were  introduced  empirically 
[13].  Our  main  theoretical  contribution  was  the  development  of  the  flow  metric 
for  using  the  ambient  metric  on  the  OFMs.  Using  the  flow  metric,  we  derived  dif¬ 
ferential  geometric  analogues  of  tangent  bundles,  vector  fields,  parallel  transport, 
curvature  etc.  When  the  IAM  is  generated  by  Lie  group  parameters,  we  showed 
that  the  OFM  framework  includes  previous  algebraic  methods  as  a  special  case. 
Moreover,  since  the  flow  neighborhood  at  each  point  is  the  entire  IAM,  we  can 
obtain  geodesics  between  any  two  images  using  flow  operators.  While  this  paper 
has  focused  on  explaining  and  extending  the  results  obtained  in  [13],  we  envision 
that  the  theory  could  make  an  on  a  large  class  of  applications  involving  image 
manifolds. 

A  clear  assumption  in  our  analysis  has  been  that  optical  flow  is  the  transport 
operator  of  choice  for  IAMs.  While  this  is  true  for  a  majority  of  IAMs  gen¬ 
erated  by  motion-induced  parameter  changes  such  as  translations,  rotations  and 
unstructured  plastic  deformations,  there  are  classes  of  IAMs  for  which  the  choice 
of  transport  operator  is  not  immediately  clear.  For  instance,  for  illumination  man¬ 
ifolds  obtained  by  variations  in  the  illumination  of  an  object,  optical  flow  may  not 
be  the  transport  operator  of  choice,  since  such  manifolds  do  not  in  general  obey 
the  brightness  constancy  requirement  needed  in  optical  flow  computations.  More¬ 
over,  in  cases  where  there  is  significant  self-occlusion  during  the  imaging  process, 
optical  flow  may  not  be  a  practical  transport  operator.  However,  by  regularizing 
the  optical  flow  computation  to  handle  occlusions  by  removing  flow  operators 
that  lead  to  undefined  motion  between  pixels,  one  can  partially  circumvent  this 
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issue,  and  the  methods  of  this  paper  can  be  profitably  applied.  However,  these 
are  aspects  of  a  more  computational  nature,  and  we  reserve  them  for  an  alternate 
forum. 

A  number  of  avenues  for  future  work  exist.  First,  our  work  hints  that  it 
should  be  possible  to  develop  additional  geometric  tools  such  as  affine  connec¬ 
tions,  holonomy,  etc.  using  flow  operators.  These  will  complete  the  I  AM  anal¬ 
ysis  toolbox.  Second,  although  our  development  has  been  specific  to  the  case 
of  IAMs  with  optical  flow,  the  basic  model  is  extensible  to  a  wide  variety  of 
signal  ensembles  with  appropriately  defined  transport  operators.  For  instance, 
a  manifold  model  for  speech  signals  has  been  proposed  in  [29]  where  appropriate 
transport  operators  and  the  analog  of  OFMs  may  involve  a  frequency  domain  ap¬ 
proach.  Once  the  “right”  transport  operator  has  been  identified  for  the  application 
in  hand,  one  can  conceivably  define  and  study  metrics  similar  to  our  flow  metric 
and  thereby  develop  analytic  tools  for  further  analysis. 
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Appendix  A.  Smoothness  and  Distance  Properties  of  OFMs 

In  this  appendix,  we  establish  the  smoothness  and  isometry  properties  of 
OFMs  associated  with  two  interesting  classes  of  IAMs:  the  affine  articulation 
manifold  and  the  pose  manifold. 

Appendix  A.l.  Affine  articulation  manifold 

Affine  articulations  are  parameterized  by  a  6-dimensional  articulation  space 
0  =  M6;  each  articulation  6  can  be  written  as  6  =  (A,  t),  with  A  e  M2x2  and 
tel2.  Any  image  belonging  to  the  IAM  can  thus  be  written  as 

J1(x)  =  Jref((A  +  I)x  +  t),  (A.l) 
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where  x  =  (x,  y )  is  defined  over  the  domain  X  =  [0, 1]  x  [0, 1]  and  where  Jre f  is  a 
reference  image.2  The  optical  flow  field  fg  associated  with  the  transport  operator 
can  be  written  as  fg(x)  =  /a, t(x)  =  Ax  +  t.  Now,  recall  that  the  OFM  at  Jre f  is 
defined  as 

Oirei  =  {fe--  0  =  (At),AGM2x2,tGM2}. 

The  linear  dependence  of  the  optical  flow  field  on  both  A  and  t  implies  that  the 
OFM,  which  is  the  collection  of  optical  flows  at  /ref,  is  infinitely  smooth.  Finally, 
noting  that  the  OFM  is  independent  of  the  reference  image,  we  can  establish  the 
following  result. 

Lemma  1.  For  affine  articulations,  the  OFM  at  any  reference  image  is  infinitely 
smooth. 


Next,  we  consider  distances  between  optical  flows  and  establish  that  the  OFM 
is  isometric. 


Lemma  2.  For  affine  articulations,  the  OFM  at  any  reference  image  is  globally 
isometric. 


Proof.  Given  two  articulations  6\  =  and  02  =  (A2)  t2),  the  Euclidean 

distance  between  them  is  given  by 


(d(fdl,  fe2)f  = 


II  fefix)  -  fgfix) 


1 2  dx 


||(v4i  -  A2)x  +  (ti  -  t2)||i rfx 


'xeT 


=  (9i-e2)Tv(e1-o2), 


where 


x 

1 


For  X  =  [0,  1]  x  [0,  1],  it  is  easily  shown  that  E  is  full-rank  and  positive  definite. 
Hence,  we  have 


d(fd1,fe2)  —  ||#i  —  ^2 1 1  £  > 

where  ||  •  ||s  is  the  Mahalanobis  (or  weighted  Euclidean)  distance  defined  using  the 
matrix  E.  This  implies  that  the  OFM  is  Euclidean  and,  hence,  globally  isometric. 

□ 


2Boundary-related  issues  are  always  an  concern  when  we  define  images  over  a  finite  domain. 
Here,  we  circumvent  this  by  assuming  that  the  regions  of  interest  are  surrounded  by  a  field  of 
zeros.  This  allows  us  to  assign  undefined  values  to  zero  and  satisfy  (A.l) 
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Appendix  A.2.  Pose  manifold 

The  pose  manifold  is  the  IAM  corresponding  to  the  motion  of  a  camera  observ¬ 
ing  a  static  scene.  It  is  well-known  that  the  articulation  space  is  6-dimensional, 
with  3  degrees  of  rotation  and  3  degrees  of  translation  of  the  camera,  i.e., 
0  =  50(3)  x  M3.  We  assume  that  the  optical  flow  is  the  (unique)  2-dimensional 
projection  of  the  motion  flow  of  the  scene  induced  due  to  the  motion  of  the  cam¬ 
era. 

Without  loss  of  generality,  we  assume  the  reference  articulation  9rei  = 
(R0,  t0)  =  (I,  0)  and  that  the  camera’s  internal  calibration  is  known  and  accounted 
for  [30].  At  the  reference  articulation  9ref,  let  the  depth  at  a  pixel  x  be  given  by 
Aref(x).  Under  an  articulation  9  =  ( Rg ,  t0),  the  optical  flow  observed  at  the  pixel 
x  is  given  by 


m(x)  =  p 


^Aref  (x)f?g 


x 

1 


P 


(A. 2) 


where  P(-)  is  a  projection  operator  such  that  P(x,  y,  z )  =  (x/z,  y/z).  Note  that 
P  is  a  well-defined  and  infinitely  smooth  function  provided  z  f  0. 

The  OFM  O  is  the  image  of  the  map  f(9)  =  {/(0)(x)  :  x  e  X},  where  X 
is  the  image  plane.  We  now  establish  the  smoothness  of  the  OFM. 

Lemma  3.  Consider  the  OFM  corresponding  to  a  pose  manifold  of  a  scene  with 
depth  map  (at  the  reference  articulation  9ref)  strictly  bounded  away  from  zero,  i.e, 
V  x,  Aref(x)  >  A  >  0.  Then  there  exists  a  neighborhood  of  9re f  where  f(9)  = 
{ _/' ( (9 )  ( x ) ;  x  G  X}  is  an  infinitely  smooth  map. 

Proof.  The  smoothness  of  f(9)  follows  from  the  smoothness  of  /(0)(x),  which 
in  turn  follows  from  the  smoothness  of  the  projection  operator  P.  Note  that 
P(x,y,z)  =  (x/z,  y/z)  is  well  defined  and  infinitely  smooth  only  if  z  f  0. 
When  all  points  in  the  scene  have  depth  bounded  away  from  zero,  then  there  is 
guaranteed  a  neighborhood  of  0  around  9re f  wherein  all  points  continue  to  have 
depth  bounded  away  from  zero.  This  ensures  that  the  projection  of  all  points  is 
well  defined,  and  hence,  f(9)  is  infinitely  smooth  in  this  neighborhood.  □ 

Note  that,  in  contrast  to  affine  articulations,  where  the  corresponding  OFM  is 
smooth  globally,  the  OFM  associated  with  the  pose  manifold  is  smooth  only  over 
a  neighborhood  of  the  reference  point. 

Next,  we  consider  distances  between  optical  flows  and  establish  that  the  OFM 
is  locally  isometric.  We  begin  by  defining  the  following  representation  for  rotation 
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matrices.  Let  u  =  (ojx,uy,ujz)  G  M3,  and  let  fiu  e  M3x3  be  the  skew- symmetric 
matrix  defined  as 


—UJy 


0 

t o7 


Ldy 

-UJZ 

0 


Noting  that  the  matrix  exponential  e  “  is  a  rotation  matrix,  we  define  our  articu¬ 
lations  as  6  =  (cc,  t)  e  M6,  with  t  =  (tx,ty,  tz). 

Before  we  state  a  formal  result,  we  need  to  introduce  two  assumptions  that 
will  help  in  simplifying  the  derivation.  First,  we  assume  that  the  imaging  model 
is  well-approximated  by  a  weak  perspective  model  [30].  In  the  weak  perspective 
model,  it  is  assumed  that  the  variations  in  the  scene  depth  are  significantly  smaller 
than  the  average  depth  of  the  scene.  In  such  a  case,  the  projection  map  P(x)  = 
(x/z,y/z)  can  be  well-approximated  by 


^wp(x)  ip^  / ^av;  V / -2av )  > 


where  zav  is  the  average  scene  depth.  As  a  consequence,  translations  along  the  pr¬ 
axis  are  unobservable  [30];  hence,  we  restrict  the  articulation  space  to  the  rotation 
of  the  camera  and  translation  along  x  and  y  axes  alone.  Second,  we  assume  that 
the  rotation  undergone  by  the  camera  is  small.  This  enables  us  to  approximate  the 
matrix-exponential  en“  ~  I  +  i\,.  As  a  consequence  of  this  assumption,  we  only 
obtain  a  local  isometry  result.  With  these  two  assumptions,  we  are  ready  to  state 
a  result  on  the  distances  between  optical  flows. 

Lemma  4.  Consider  the  pose  manifold  under  the  assumption  of  the  weak  per¬ 
spective  imaging  model.  Then  the  OFM  is  locally  isometric. 

Proof.  Without  loss  of  generality,  we  denote  our  reference  articulation  as  0r,,(  =  0, 
which  corresponds  to  identity  rotation  matrix  and  null  translation  vector.  We  are 
interested  in  the  distance  between  two  optical  flow  fields  f{9 1)  and  f(92).  Under 
our  assumptions  of  weak  perspective  imaging  model  and  small  rotation,  we  can 
compute  the  distance  between  the  optical  flows  to  be 


f(9 i)(x)  -  f(d 2)(x) 


1 

Aw 

1 

Aw 


( ref  Pi 


V 


x 

1 


+ 1] 


P 


wp 


(  Aref-^2 


V 


X 

1 


+  t' 


(  ^ref  ^1  ,y)  ^l,a?)  (  'Vef  (^2 ,xV  ^2 ,y) 

(^ref (^l,a^  "I-  ^l,y)  (^ref (^2,®*^  ^2,z)  ^2 ,y) 

^ref  (^1,#  (j^21x)V  "“I”  ^ref(^l,y  ^2 ,y)  “1“  l,x  ^2,x) 

Aref(^l,x  ^2,x)^  ^ref(^l,2  ^2,z)  “I-  (^1  ,y  ^2 ,?/) 


33 


Notice  that  both  tljZ  and  t2z  are  absent  in  the  expression  above.  As  noted  above, 
this  is  due  to  the  assumption  of  weak  perspective  imaging  model  which  makes  the 
translation  of  the  camera  along  the  z-axis  unobservable.  A  key  observation  is  that 
the  articulation  parameters  9\  =  (aq,  ti)  and  02  =  (cu2,  t2)  can  be  expressed  as  a 
function  that  is  linear  in  the  expression  above.  Hence, 


f(0 0(x)  -  f(e 2)(x)  =  A(x) 

Finally,  summing  over  x  e  X,  we  obtain 


tUi  -  0J2 

ti  — t2 


d(f(e  i),/(02))2  =  (0i-02) 


AT(x)A(x)dx  (0!  -  02) 


'xeA” 


Hence,  the  OFM  is  locally  isometric. 


□ 


References 

[1]  D.  L.  Donoho,  C.  C.  Grimes,  Image  manifolds  which  are  isometric  to  eu¬ 
clidean  space,  J.  of  Math.  Imaging  and  Vision  23  (2005)  5-24. 

[2]  M.  B.  Wakin,  D.  L.  Donoho,  H.  Choi,  R.  G.  Baraniuk,  The  multiscale  struc¬ 
ture  of  non-differentiable  image  manifolds,  in:  Proc.  SPIE  Wavelets  XI, 
2005. 

[3]  R.  R.  Coifman,  S.  Lafon,  A.  B.  Lee,  M.  Maggioni,  F.  Warner,  S.  Zucker, 
Geometric  diffusions  as  a  tool  for  harmonic  analysis  and  structure  definition 
of  data:  Diffusion  maps  102  (21)  (2005)  7426-7431. 

[4]  B.  J.  Culpepper,  B.  A.  Olshausen,  Learning  transport  operators  for  image 
manifolds,  in:  Adv.  in  Neural  Info.  Proc.  Sys.,  2009. 

[5]  J.  Tenenbaum,  V.  de  Silva,  J.  Landford,  A  global  geometric  framework  for 
nonlinear  dimensionality  reduction,  Science  290  (5500)  (2000)  2319-2323. 

[6]  S.  Roweis,  L.  Saul,  Nonlinear  dimensionality  reduction  by  locally  linear  em¬ 
bedding,  Science  290  (5500)  (2000)  2323-2326. 

[7]  M.  Belkin,  P.  Niyogi,  Laplacian  eigenmaps  and  spectral  techniques  for  em¬ 
bedding  and  clustering,  in:  Adv.  in  Neural  Info.  Proc.  Sys.,  2001. 


34 


[8]  Y.  Ma,  S.  Soatto,  J.  Kosecka,  S.  Sastry,  An  Invitation  to  3-D  Vision:  From 
Images  to  Geometric  Models,  Springer- Verlag,  2005. 

[9]  M.  P.  do  Carmo,  Riemannian  Geometry,  Birkhauser,  1992. 

[10]  X.  Miao,  R.  P.  N.  Rao,  Learning  the  lie  groups  of  visual  invariance,  Neural 
computation  19  (10)  (2007)  2665-2693. 

[11]  D.  Lin,  E.  Grimson,  J.  Fisher,  Learning  visual  flows:  a  lie  algebraic  ap¬ 
proach,  2009. 

[12]  O.  Tuzel,  F.  Porikli,  P.  Meer,  Learning  on  lie  groups  for  invariant  detection 
and  tracking,  in:  IEEE  Conf.  Comp.  Vision  and  Patt.  Recog.,  2008. 

[13]  A.  C.  Sankaranarayanan,  C.  Hegde,  S.  Nagaraj,  R.  G.  Baraniuk,  Go  with  the 
flow:  Optical  flow-based  transport  for  image  manifolds,  in:  Allerton  Conf. 
on  Comm.,  Control  and  Computing,  2011. 

[14]  R.  Li,  R.  Chellappa,  Aligning  spatio-temporal  signals  on  a  special  manifold, 
in:  Euro.  Conf.  on  Comp.  Vision,  2010. 

[15]  M.  I.  Miller,  L.  Younes,  Group  actions,  homeomorphisms,  and  matching:  a 
general  framework,  Inti.  J.  of  Comp.  Vision  41  (1-2)  (2001)  61-84. 

[16]  B.  K.  P.  Horn,  B.  G.  Schunck,  Determining  optical  flow,  Artificial  intelli¬ 
gence  17  (1-3)  (1981)  185-203. 

[17]  D.  Sun,  S.  Roth,  M.  J.  Black,  Secrets  of  optical  flow  estimation  and  their 
principles,  in:  IEEE  Conf.  Comp.  Vision  and  Patt.  Recog.,  2010. 

[18]  T.  Brox,  J.  Malik,  Large  displacement  optical  flow:  descriptor  matching  in 
variational  motion  estimation,  IEEE  Trans.  Pattern  Anal.  Mach.  Intell.  33  (3) 
(2011)  500-513. 

[19]  H.  Karcher,  Riemannian  center  of  mass  and  mollifier  smoothing,  Comm,  on 
Pure  and  Appl.  Math.  30  (5)  (1977)  509-541. 

[20]  T.  F.  Cootes,  C.  J.  Taylor,  D.  H.  Cooper,  J.  Graham,  Active  shape  models- 
their  training  and  application,  Comp,  vision  and  image  understanding  61(1) 
(1995) 38-59. 


35 


[21]  T.  F.  Cootes,  G.  J.  Edwards,  C.  J.  Taylor,  Active  appearance  models,  IEEE 
Trans.  Pattern  Anal.  Mach.  Intell.  23  (6)  (2001)  681-685. 

[22]  J.  Kaplan,  D.  Donoho,  The  morphlet  transform:  A  multiscale  representation 
for  diffeomorphisms,  in:  Workshop  on  Image  Registration  in  Deformable 
Environments,  2006. 

[23]  D.  Beymer,  T.  Poggio,  Image  representations  for  visual  learning,  Science 
272  (5270) (1996)  1905-1909. 

[24]  N.  Jojic,  B.  J.  Frey,  Learning  flexible  sprites  in  video  layers,  in:  IEEE  Conf. 
Comp.  Vision  and  Patt.  Recog.,  2001. 

[25]  S.  A.  Nene,  S.  K.  Nayar,  H.  Murase,  Columbia  object  image  library  (COIL- 
100),  T.R.  No.  CUCS-006-96,  Dept.  Comp.  Science,  Columbia  University. 

[26]  M.  E.  Munich,  P.  Perona,  Continuous  dynamic  time  warping  for  translation- 
invariant  curve  alignment  with  applications  to  signature  verification,  in:  Inti. 
Conf.  on  Comp.  Vision,  1999. 

[27]  K.  Wang,  T.  Gasser,  Alignment  of  curves  by  dynamic  time  warping,  Annals 
of  Statistics  25  (3)  (1997)  1251-1276. 

[28]  A.  Bosco,  A.  Bruna,  S.  Battiato,  G.  Puglisi,  Digital  video  stabilization 
through  curve  warping  techniques,  IEEE  Trans,  on  Consumer  Electronics 
54  (2) (2008)  220-224. 

[29]  A.  Jansen,  P.  Niyogi,  Intrinsic  fourier  analysis  on  the  manifold  of  speech 
sounds,  in:  ICASSP,  2006. 

[30]  R.  Hartley,  A.  Zisserman,  Multiple  view  geometry,  Cambridge  university 
press,  2000. 


36 


